$$ Project \ 4: \ Feature \ Engineering, \ Model \ Selection, \ Model \ Tuning \ $$¶
$$ Concrete \ Compressive \ Strength \ Prediction $$¶
$$ (A \ Regression \ Problem) $$¶
$Objective:$ To predict the Concrete Compressive Strength using the data available in file concrete_data.xls. Apply Feature Engineering and Model Tuning to obtain $80\% \ to \ 95\% \ of \ R^2 \ Score$.¶
$Derived \ Project \ Conclusion$: Test Data SCORE: 0.9161131402242251 ***(91.61%)***: On the UNSEEN / Reserved Test Data : BEST Final SCORE on high end of Goal Range: 80% to 95% : Objective Achieved !!!¶
$Resources \ Available:$ The data for this project is available in file https://archive.ics.uci.edu/ml/machine-learning-databases/concrete/compressive/. The same has been shared along with the course content.¶
Given Information AS Per Project Doc & Data Resource URL Above:
- ### $Data \ Characteristics$: Data is in ***RAW Form (NOT Scaled).*** The actual concrete compressive strength (MPa) for a given mixture under a specific age (days) was determined from laboratory.
- ### $Attribute \ Information$: The order of this listing corresponds to the order of numerals along the rows of the database:
| No. | Name | Meaning | Data Name | Short Data Name | Analytic Type | Data Type | Unit Of Measure | Variable Type | Description / Comments |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Cement | An Ingredient | cement | cement | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 2 | Blast Furnace Slag | An Ingredient | slag | slag | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 3 | Fly Ash | An Ingredient | ash | ash | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 4 | Water | An Ingredient | water | water | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 5 | Superplasticizer | An Ingredient | superplastic | splast | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 6 | Coarse Aggregate | An Ingredient | coarseagg | corse | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 7 | Fine Aggregate | An Ingredient | fineagg | fine | Quantitative | Numeric | Kg/m3 mix | Input | Predictor Variable |
| 8 | Age | Days from Casting | age | age | Quantitative | Numeric | Day (1...365) | Input | Predictor Variable |
| 9 | Concrete Compressive Strength | Measure of Strength | strength | mpa | Quantitative | Numeric | MPa (Mega Pascal) | OUTPUT | PREDICTED Variable |
- Added By MSB (Project Coder/Developer) Above^: New Info added to the table above: Short Data Name , Data Type , Comments
- ### $The \ Concrete \ Compressive \ Strength \ Prediction$: It is a Multi Variate Regression Problem having:
- Eight (8) Input (Predictors) variables (Data Columns)
- One (1) OUTPUT (PREDICTED) variable (Data Column)
- Nine (9) Total Attributes (Data Columns)
- 1,030 Total Observarions (Data Rows)
- No (Zero) Missing Attributes Values (Data Columns Values)
Project Rubric / Scoring Guide: Concrete Compression Strength Prediction¶
Points / Marks By Criteria:¶
- 10 = Univariate Analysis
- 10 = Bivariate Analysis
- 10 = Feature Engineering
- 15 = Modelling
- 15 = Hyper Parameter Tuning
- 60 = Total Points
Project Steps & Tasks: Deliverables: Concrete Compression Strength Prediction¶
*D1: Exploratory Data Quality Report reflecting the following:*¶
###### D1.1: Univariate Analysis: 10 Marks:
Data types and description of the independent attributes which should include (name, meaning, range of values observed, central values (mean and median), standard deviation and quartiles, analysis of the body of distributions / tails, missing values, outliers.###### D1.2: Bi-variate Analysis: 10 Marks:
Analyze Among Predictor Variables and Between Predictor & Target Columns. Comment on your findings in terms of their Relationship and Degree of Relation if any. Visualize the analysis using Boxplots and Pair Plots with Histograms or Density Curves.###### D1.3: Feature Engineering Techniques: 10 Marks:
- $D1.3.a:$ Identify opportunities (if any) to Extract a New Feature from existing features and/or Drop a Feature (if required).
- $D1.3.b:$ Get data model ready and do a train test split.
- $D1.3.c:$ Decide on complexity of the model, should it be Simple Linear Model in terms of Parameters or would it be a Quadratic or Higher degree.
*D2: Creating the Model and Tuning it:*¶
- ###### D2.1: Algorithms: 15 Marks:
Use the Algorithms that you think will be suitable for this project (at least 3 algorithms). Use ***Kfold Cross Validation*** to evaluate model performance. Use appropriate metrics and make a DataFrame to compare models w.r.t their metrics. - ###### D2.2: Techniques: 15 Marks:
Employ Techniques to squeeze that extra performance out of the model without making it over fit. Use ***Grid Search or Random Search*** on any of the two models used above. Make a DataFrame to compare models after hyperparameter tuning and their metrics as above.
In [1422]:
# v====== Standard Libraries Begin ======v #
import warnings
warnings.filterwarnings('ignore')
import numpy as np # Numerical Python libraries
# random_state = np.random.RandomState(0) # From Mukesh Rao. MSB: Do we need to do this? Working ok without it.
import pandas as pd # to handle data in form of rows and columns
import pandas_profiling
import pylab as pl # Mukesh Rao
import seaborn as sns # Data visualization for statistical graphics
import matplotlib.pyplot as plt # Data visualization for Ploting
from sklearn.svm import SVC # M.Rao; SVC = Support Vector Classification
from sklearn import metrics, svm # "svm" = Support Vector Machine > MRao; For Lin/Log Regr, DTree
from sklearn.impute import SimpleImputer
from sklearn.utils import resample, shuffle # "shuffle"=> Mukesh Rao; Bagging Sample data set creation
from sklearn.model_selection import train_test_split, KFold, cross_val_score, LeaveOneOut, GridSearchCV, RandomizedSearchCV # Lin/LogR, DTree
from sklearn.pipeline import Pipeline, make_pipeline # M.Rao
from sklearn.neighbors import KNeighborsClassifier # MRao
# For Linear Dimensionality (Cols/Attributes) Reduction to a Lower dimensional space (eg: reduce 15 cols to 2 cols):
from sklearn.decomposition import PCA # "Principal Component Analysis" for "Singular Value Decomposition" (SVD)
# ClusterCentroids=Cluster based UNDERsampling, TomekLinks=Under sampling by Deleting nearest majority neighbor/similar rows
from imblearn.under_sampling import RandomUnderSampler, ClusterCentroids, TomekLinks
from imblearn.over_sampling import SMOTE # Over sampler
from imblearn.combine import SMOTETomek # OVER / UP Sampling followed by UNDER / DOWN Sampling
from mlxtend.feature_selection import SequentialFeatureSelector as sfs # For Features selection
from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs # For Plotting
# ====== For Linear Regression ======
from scipy.stats import zscore, pearsonr, randint as sp_randint
from category_encoders import OrdinalEncoder
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, MinMaxScaler, PolynomialFeatures, binarize, LabelEncoder, OneHotEncoder # M.Rao
from sklearn.linear_model import LogisticRegression, LinearRegression, Lasso, Ridge
# Import Linear Regression machine learning library:
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score # LinRegr
import statsmodels.api as sm # For OLS Summary in Linear Regression
import statsmodels.formula.api as smf # For OLS Summary in Linear Regression
from yellowbrick.regressor import ResidualsPlot
from yellowbrick.classifier import ClassificationReport, ROCAUC
# ====== For Logistic Regression ======
from sklearn.metrics import confusion_matrix, recall_score, precision_score, accuracy_score
from sklearn.metrics import f1_score, roc_curve, roc_auc_score, classification_report, auc # Mukesh Rao
# ====== For Decision Tree ======
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, export_graphviz
# from sklearn.externals.six import StringIO # Discontinued in Scikit Version 0.23 (available only upto Ver 0.22)
import pydotplus as pdot # to display decision tree inline within the notebook
import graphviz as gviz
# DTree does not take strings as… # … input for the model fit step....
from sklearn.feature_extraction.text import CountVectorizer
# ======= For Ensemble Techniques =======
from sklearn.ensemble import BaggingClassifier, RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier
from sklearn.ensemble import BaggingRegressor, RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor
# ======= Set default style ========
# Multiple output displays per cell
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
from IPython.display import Image, Markdown
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:98% !important; }</style>")) # Increase cell width
# ===== Options =====
pd.options.display.float_format = '{:,.2f}'.format # Remove scientific notations to display numbers with 2 decimals
pd.set_option('display.max_columns', 100) # Max df cols to display set to 100.
pd.set_option('display.max_rows', 50) # Max df rows to display set to 50.
# pd.set_option('display.max_rows', tdf.shape[0]+1) # just one row more than the total rows in df
# Update default style and size of charts
plt.figure(figsize=(12,8))
plt.style.use('ggplot') # plt.style.use('classic') ??
plt.rcParams['figure.figsize'] = [10, 8]
sns.set_style(style='darkgrid')
%matplotlib inline
import pickle # For model export
from os import system # For system (eg MacOS, etc) commands from within python
# ====== Standard Libraries End ======^ #
Out[1422]:
In [58]:
# Read & Load the input Datafile into dataset frame: Concrete DataFrame:
cdf = pd.read_csv('concrete.csv')
cdf
Out[58]:
In [65]:
# My Housekeeping: Incremental DF Data Backup 0 as of now:
cdf0 = cdf.copy() # Original Df
cdf.to_csv('cdf0.csv') # Also export as .csv file to disk
! ls -l cdf*
# Verify backup copy
cdf0.shape, type(cdf0)
cdf0.sample(7)
Out[65]:
Out[65]:
In [66]:
# My Housekeeping: Rename column names for convenience, meaningfulness and intuitiveness:
cdf.rename(columns={'superplastic': 'splast', 'coarseagg': 'corse', 'fineagg': 'fine', 'strength': 'mpa'},
inplace=True, errors='raise')
cdf.sample(6)
Out[66]:
In [72]:
# My Housekeeping: Incremental DF Data Backup 1 as of now:
cdf1 = cdf.copy() # Modified Df: Changed Columns Names to shorten: 'splast', 'corse', 'fine', 'mpa'
cdf1.to_csv('cdf1.csv') # Also export as .csv file to disk
! ls -l cdf*
# Verify backup copy
cdf1.shape, type(cdf1)
cdf1.sample(6)
Out[72]:
Out[72]:
D1.1: Univariate Analysis: Begins below: vvv¶
Data types & description of the independent attributes which should include (name, meaning, range of values observed, central values (mean and median), standard deviation and quartiles, analysis of the body of distributions / tails, missing values, outliers.
Note: Data Name, Type, Meaning, Description has been tabulated above in Header Setion at the Top^^^¶
In [68]:
profile = pandas_profiling.ProfileReport(cdf)
profile
Out[68]:
In [153]:
### My Housekeeping: Jupyter Code File Incremental Backup 1 Tue.Jun.30 2:47am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 1")
# ! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 1.ipynb"
! ls -l Project*.ipynb
Out[153]:
In [569]:
Markdown('### * DF Head & Tail rows:')
cdf
Markdown('### * DF Random Sample rows:')
cdf.sample(7)
Markdown('### * DF Shape: Number of (Rows, Columns), DF Type:')
cdf.shape, type(cdf)
Markdown('### * DF Info with: Column Names & Data Types')
cdf.info()
Markdown('### * DF Stats for Numerical value Cols: Range (Min & Max), Central values (Mean), Std.D, Quartiles')
cdf.describe()
Markdown('### * DF Stats for Numerical value Cols: Central values: MEDIAN which are not in Standard "Describe" above^:')
cdf.median()
Markdown('### * DF Stats for All Cols: Central values: MODE which are not in Standard "Describe" above^:')
cdf.mode()
Markdown(""" ### * $ \ {dup}$ = Duplicate <u>Columns</u> Based on ALL Rows """.format(dup=cdf.T.duplicated().sum()))
Markdown(""" ### * ${dup}$ = Duplicate <u>Rows</u> Based on ALL 9 Columns """.format(dup=cdf.duplicated().sum()))
Markdown('### * DF Duplicated <u>Rows</u> Based on CERTAIN "Mixture Ingredients" <u>Columns</u>:')
print(' ', cdf.iloc[:,:8].duplicated().sum(),
'= Dup. Rows for first 8 Cols: cement, slag, ash, water, splast, corse, fine, age. (Except: "mpa")')
print('', cdf.iloc[:,:7].duplicated().sum(),
'= Dup. Rows for first 7 Cols: cement, slag, ash, water, splast, corse, fine. (Except: "age" & "mpa")')
print(' ', cdf.duplicated(['cement', 'slag', 'ash', 'water', 'splast', 'corse', 'fine', 'mpa']).sum(),
'= Dup. Rows for first 7 & "mpa" Cols: cement, slag, ash, water, splast, corse, fine, mpa. (Except: "age")')
Markdown('### * DF Unique Values for All Columns:')
cdf.nunique()
Markdown('### * DF Null values for All Columns:')
cdf.isna().sum()
Markdown('### * DF NON Numeric Values in Numerical Columns:')
cdf[~cdf.select_dtypes(include='number').applymap(np.isreal).all(1)].count() # With "~" for NOT Real Numbers (Non Numeric)
Markdown('### * DF Zero Values in Numerical Columns:')
(cdf.select_dtypes(include='number') == 0).sum()
Markdown('### * DF Negative (-ve) Values in Numerical Columns:')
(cdf.select_dtypes(include='number') < 0).sum()
Markdown('### * DF Skewness for All Numerical Columns:')
cdf.skew()
Markdown('###### * Note: Skew Categories (Arbitrary): ')
print('''If distribution Skewness is between following ranges then Skewness for Column is...:
* High : < −1 OR > +1 Asymmetric Col : age
* Moderate : −1 & −0.5 OR +0.5 & +1 Asymmetric Cols: cement, slag, ash, splast
* Low : −0.5 & +0.5 Asymmetric Col : mpa
* V.Low : −0.25 & +0.25 Symmetric Col : fine
* No Skew : 0.0 OR near -0.0+ Symmetric Cols: water, corse
''')
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
Out[569]:
In [154]:
### My Housekeeping: Jupyter Code File Incremental Backup 2 Sun.Jul.05 9:49pm ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 2")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 2.ipynb"
! ls -l Project*.ipynb
Out[154]:
In [571]:
# Indetify Outlier Values in All Numerical Columns:
Markdown('### * DF Number of Outliers for Numerical Columns: $ \ \ Low = Q1 - (IQR * 1.5) $; $ \ \ High = Q3 + (IQR *1.5 $)')
for col in cdf.select_dtypes(include='number'):
q1 = cdf[col].quantile(.25)
q3 = cdf[col].quantile(.75)
otr = (q3 - q1) * 1.5
otl = int(q1 - otr)
oth = int( q3 + otr)
otls = (cdf[col] < otl).sum()
oths = (cdf[col] > oth).sum()
print('*', col)
print(' ', str(otls).rjust(2, ' '), 'outliers Under Low End', str(otl).rjust(5, ' '))
print(' ', str(oths).rjust(2, ' '), 'outliers Over High End', str(oth).rjust(5, ' '))
print()
Out[571]:
In [410]:
# Histogram & Density of Entire Dataset: UnScaled & Scaled (zscore): Visual Distribution of DF values:
p = plt.figure(figsize=(20,7))
p = plt.subplot(1, 2, 1)
g = sns.distplot(cdf)
p = plt.subplot(1, 2, 2)
g = sns.distplot(cdf.corr())
In [346]:
# Individual Histogram of ALL 9 Numerical Columns: Visual Distribution of column values:
# catch plt sns outputs in some variables ("p" & "g") to supress informational output
p = plt.figure(figsize=(20, 20))
pos = 1
for col in cdf.columns:
p = plt.subplot(3, 3, pos)
g = sns.distplot(cdf[col])
pos += 1
D1.1 Comments / Conclusions / Recommendations:¶
- Very many Zero values in 3 columns: slag = 471, (46%), ash = 566, (55%), splast = 379, (37%) = Impute / Scale / Drop "ash"? = NO. This are VALID ZERO valuea. Leave them as it is = Per Domain/Academic Support: These can be zeros and are VALID
- Not many Duplicate Rows: Only 25 (2.43%) = Delete/Drop.
Skewness: Scale
- High : age
- Moderate : cement, slag, ash, splast
- Low : mpa
- V.Low : fine
- No Skew : water, corse
Very few outliers: Impute / Scale.
- cement, ash, corse: No (zero) Outliers
- slag (rare): 2 outliers (Over High Limit)
- water (medium): 9 outliers ( 5 Under Low; 4 Over High Limits)
- splast (medium): 10 outliers (Over High Limit)
- fine (rare) : 5 outliers (Over High Limit)
- age (high) : 59 outliers (Over High Limit)
- mpa (medium) : 9 outliers (Over High Limit)
D1.1 ^^^Completed & Delivered As Per Above Exhibits: Univariate Analysis Ends here: ^^^¶
- Displayed: Data Types, Names, Description, Meaning, Data Profile, Data Example Rows (Head, Tail, Random Sample)
- Generated: Date Values Range (Min/Max, Value Counts), Central Values (Mean, Median, Mode), Standard Deviation, Quartiles
- Presented: Visual / Graphical Data Distribution and Tails for Numerical Values
- Identified: Zero & Negative Values, Missing (Null & Unknown/Unspecified) Values, Outliers
- Conclusion: Comments & Recommendations provided above^
##### Note: Data Name, Type, Meaning, Description has been tabulated in the Header Section at the Top^^^
In [379]:
### My Housekeeping: Jupyter Code File Incremental Backup 3 Mon.Jul.06 7:39am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 3")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 3.ipynb"
! ls -l Project*.ipynb
Out[379]:
D1.2: Bi Variate Analysis Starts below: vvv¶
Analyze Among Predictor Variables and Between Predictor & Target (Predicted) Columns. Comment on your findings in terms of their Relationship and Degree of Relation if any. Visualize the analysis using Boxplots and Pair Plots with Histograms or Density Curves.
In [613]:
Markdown('##### * PairPlot BiVariate Study among Predictor variables & with Predicted/Target column, with Density Curves (diag_kind="kde")')
g = sns.pairplot(cdf, diag_kind='kde')
Out[613]:
In [897]:
Markdown('##### * Individual Box Plots of ALL 9 Numerical Columns: Visuals For Quartiles, Middle & Outlier values:')
p = plt.figure(figsize=(20, 20))
pos = 1
for col in cdf.columns:
p = plt.subplot(3, 3, pos)
g = sns.boxplot(cdf[col])
pos += 1
Out[897]:
In [818]:
# "Age" has 14 (limited) "bin like" non continous numerical values, though it can have continous values from 1 to 365
# Hence we can use BoxPlot & PointPlot to do BiVariate study between predictor "age" col and "mpa" col (predicted/target):
Markdown('### <center> * BoxPlot & PointPlot : BiVariate Study of "age" & "mpa" * </center>')
p = plt.figure(figsize=(20, 20))
g = sns.boxplot('age', 'mpa', data=cdf)
g = sns.pointplot(cdf.age, cdf.mpa)
Out[818]:
In [618]:
# Additional Exhibit 1:
Markdown('### * Correlation Matrix:')
cdf.corr()
Markdown('### * Correlation HeatMap Matrix:')
p = plt.figure(figsize=(11, 7))
g = sns.heatmap(cdf.corr())
Out[618]:
Out[618]:
Out[618]:
In [666]:
# Additional Exhibit 2:
# Density, Histogram, Scatter plots of All Attributes Interactions:
# Different Upper & Lower triangles: eg. Scatter & KDE (Density)
Markdown('### * PairGrid BiVariate Study among Predictor variables & with Predicted/Target column')
Markdown('* <u> Upper Half</u> : LINEAR REGRESSION Fitted line thru Scatter Plot')
Markdown('* <u> Lower Half</u> : Kernel Densities among Atributes')
g = sns.PairGrid(cdf)
g = g.map_upper(sns.regplot)
g = g.map_lower(sns.kdeplot)
g = g.map_diag(plt.hist, lw=2)
plt.show()
Out[666]:
Out[666]:
Out[666]:
In [914]:
# Additional Exhibit 3:
Markdown('##### * BiVariate Study of Predicted/Target "mpa" (y) column with remaining 8 (X) Predictor attributes:')
Markdown('''* Regression Line Fitted thru Scatter Plot For:
1=Linear, 2=Quadratic, 3=Cubic. Using regplot(X, y) for order=1,2,3, Confidence Interval ci = 95 (line shadows)''')
p = plt.figure(figsize=(20, 20))
pos = 1
for col in cdf.columns:
p = plt.subplot(3, 3, pos)
g = sns.regplot(col, 'mpa', data=cdf, order=1)
g = sns.regplot(col, 'mpa', data=cdf, order=2)
g = sns.regplot(col, 'mpa', data=cdf, order=3)
p = plt.legend(['Linear','Quadratic','Cubic'], loc="best")
pos += 1
Out[914]:
Out[914]:
In [820]:
### My Housekeeping: Jupyter Code File Incremental Backup 4 Wed.Jul.08 12:55am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 4")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 4.ipynb"
! ls -l Project*.ipynb
Out[820]:
D1.2 Comments / Conclusions / Recommendations:¶
Relationship & Degree: Correlations: None are HIGH (-0.80+): Column(s) DROP = None
- High : None
- Significant : splast & water = -0.66 (Negative Correlation)
- High Medium : cement & map = +.50 (Positive Correlation)
- Low Medium : water & fine = -.45; cement & ash = -.40 (Negative Correlations)
- Low/Notable : ash & splast = +.38; splast & map = +.37; age & mpa = +.33 (Positive Correlation)
Data Dispersion: Multiple Gaussians (Humps) in Distributions: SCALE Data = Yes
- Minimal : mpa, cement
- Moderate: water, corse, fine
- Significant: slag, ash, splast, age
Data Dispersion: Regression Estimate: SCALE Data = Yes; FE/MT = Polynomials, Adv.Regr.Models, HyperTune
- As apparent from the above Regression Plot (regplot, order=1,2,3) between "mpa" (Target) & 8 Predictor cols, that we are UNABLE to fit ANY regression line thru the Data the way its Dispersed (spread out) with even a reasonably accuracy, because the data points are scattered so far out from the regression lines for Linear, Quadratic, Cubic relationships (polynomials).
D1.2 ^^^ Completed & Delivered As Per Above Exhibits: BiVariate Analysis Ends here: ^^^¶
Visialized:
- PairPlots: Displayed with Density
- BoxPlots: Displayed for 1. For each Column. 2. With "age" & "mpa" with overlayed PointPlot ("age" & "mpa")
- Corelation & Heatmap: Displayed
- PairGrid: Displayed with Histogram and Kernel Densities (lower half, "kdeplot") & Linear Regression (upper half, "regplot")
- RegPlot: Displayed Regression Plots for orders/degrees: 1=Linear, 2=Quadratic, 3=Cubic: For each Predictor column with "mpa" Predicted/Target column
Analyzed: Interation / Relationship among Predictor Variables AND between Predictors & Predicted/Target ("mpa") column. Observations and Findings on Relationship & Degree Relation provided above^
- Comments: Conclusions & Recommendations provided above^
In [933]:
### My Housekeeping: Jupyter Code File Incremental Backup 5 Wed.Jul.08 11:23pm ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 5")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 5.ipynb"
! ls -l Project*.ipynb
Out[933]:
vvv D1.3: Feature Engineering Techniques: Begins Below vvv :¶
- $D1.3.a:$ Identify opportunities (if any) to Extract a New Feature from existing features and/or Drop a Feature (if required).
- $D1.3.b:$ Get data model ready and do a train test split.
- $D1.3.c:$ Decide on complexity of the model, should it be Simple Linear Model in terms of Parameters or would it be a Quadratic or Higher degree.
In [945]:
# Drop Duplicate Rows (25) as determined earlier in UniVariate section:
cdf1.shape # before
cdf.drop_duplicates(keep='first', inplace=True)
cdf.shape # after
print('Rows Dropped:', len(cdf1)-len(cdf))
Out[945]:
Out[945]:
In [946]:
# My Housekeeping: Incremental DF Data Backup 2 as of Thu.Jul.09 3:29am :
cdf2 = cdf.copy() # Modified Df: Dropped 25 Duplicated Rows
cdf2.to_csv('cdf2.csv') # Also export as .csv file to disk
! ls -l cdf*
# Verify backup copy
cdf2.shape, type(cdf2)
cdf2.sample(5)
Out[946]:
Out[946]:
Outlers Imputation Strategies Evaluated for 99 Total outliers in the DF:¶
- Methods NOT Used: Because it produced More / New Outliers after Imputations. (Tried Separately, code clutter removed):
- Deleting Rows with Outliers Imputing Outliers with MEDIAN values Imputing Outliers with -(IQR * 1.5)+ values
- Method USED: * Imputing Outliers with Q1 / Q3 values: : This method eliminating ALL existing outliers without creating more / new ones.
In [1063]:
# Impute All Outliers based on IQR Method with respective columns' Q1 or Q3 values:
Markdown('### * DF Outliers Imputations with Q1 (Under Low End) or Q3 (Over High End) Values for All Numerical Columns: $ \ \ Low = Q1 - (IQR * 1.5) $; $ \ \ High = Q3 + (IQR *1.5 $)')
for col in cdf.select_dtypes(include='number'):
q1 = cdf[col].quantile(.25)
q3 = cdf[col].quantile(.75)
otr = (q3 - q1) * 1.5
otl = round(q1 - otr,2)
oth = round(q3 + otr,2)
otls = (cdf[col] < otl).sum()
oths = (cdf[col] > oth).sum()
cdf[col] = np.where((cdf[col] < otl), q1, cdf[col])
cdf[col] = np.where((cdf[col] > oth), q3, cdf[col])
print('*', col, ':', otls+oths, 'values imputed with Q1, Q3 value:', otl, oth)
print(' ', str(otls).rjust(2, ' '), 'outliers Under Low End', str(otl).rjust(5, ' '))
print(' ', str(oths).rjust(2, ' '), 'outliers Over High End', str(oth).rjust(5, ' '))
print()
Out[1063]:
In [1076]:
# Post Imputation: Indetify New Outlier Values in All Numerical Columns:
Markdown('### * DF New Outliers After Imputations for Numerical Columns: NO OUTLIERS: \n $ \ \ Low = Q1 - (IQR * 1.5) $; $ \ \ High = Q3 + (IQR *1.5 $)')
for col in cdf.select_dtypes(include='number'):
q1 = cdf[col].quantile(.25)
q3 = cdf[col].quantile(.75)
otr = (q3 - q1) * 1.5
otl = round(q1 - otr,2)
oth = round(q3 + otr,2)
otls = (cdf[col] < otl).sum()
oths = (cdf[col] > oth).sum()
print('*', col)
print(' ', str(otls).rjust(2, ' '), 'outliers Under Low End', str(otl).rjust(5, ' '))
print(' ', str(oths).rjust(2, ' '), 'outliers Over High End', str(oth).rjust(5, ' '))
print()
Out[1076]:
In [1150]:
# Verify Outliers with BoxPlot:
Markdown('##### * Verify Outliers with BoxPlot: NO OUTLIERS:')
p = plt.figure(figsize=(20, 7))
g = cdf.boxplot()
Out[1150]:
In [1081]:
# My Housekeeping: Incremental DF Data Backup 3 as of Thu.Jul.09 8:01am :
cdf3 = cdf.copy() # Modified Df: Imputed 99 Outlier values to Q1 and/or Q3 column values respectively as applicable.
cdf3.to_csv('cdf3.csv') # Also export as .csv file to disk
! ls -l cdf*
# Verify backup copy
cdf3.shape, type(cdf3)
cdf3.sample(5)
Out[1081]:
Out[1081]:
In [1082]:
### My Housekeeping: Jupyter Code File Incremental Backup 6 Thu.Jul.09 08:04am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 6")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 6.ipynb"
! ls -l Project*.ipynb
Out[1082]:
In [1326]:
cdf1
Out[1326]:
In [1499]:
# Prepare data for split: Create X, y (Predictor, Predicted) datasets:
# X = cdf1.copy() # Contains: 25 duplicates; 99 Outliers, i.e. Before removing dups & outliers
X = cdf.copy() # Dups & Outliers Removed
y = X.pop('mpa')
# Split df data into 3 datasets: Train, Test, Validate:
X_trn, X_tst, y_trn, y_tst = train_test_split(X, y, test_size=0.2, random_state=3)
X_trn, X_val, y_trn, y_val = train_test_split(X_trn, y_trn, test_size=0.25, random_state=3)
In [1500]:
X.shape, y.shape, X_trn.shape, X_tst.shape, X_val.shape, y_trn.shape, y_tst.shape, y_val.shape
type(X_trn), type(X_tst), type(X_val), type(y_trn), type(y_tst), type(y_val)
Out[1500]:
Out[1500]:
In [1501]:
lrm = LinearRegression()
lrm.fit(X_trn, y_trn)
Out[1501]:
In [1502]:
print('\n* UnScaled Data: Coefficients:', lrm.coef_)
print('\n* UnScaled Data: Intercept:', lrm.intercept_)
Markdown('###### * UnScaled Data: $R^2$ Score for: In Sample & Out Of Sample (Validaton dataset):')
print(' * In Sample R^2:', lrm.score(X_trn, y_trn), 'Out Of Sample R^2:', lrm.score(X_val, y_val))
print(' * Root Mean Square Error RMSE:', mean_squared_error(y_val, lrm.predict(X_val))**0.5)
# print(lrm.summary())
Out[1502]:
In [1503]:
# print('Predict y on Validation dataset:', lrm.predict(X_val).reshape(-1,))
In [1504]:
std_sclr = StandardScaler()
X_trn = std_sclr.fit_transform(X_trn)
X_val = std_sclr.fit_transform(X_val)
X_trn = pd.DataFrame(X_trn)
X_val = pd.DataFrame(X_val)
y_trn = pd.DataFrame(y_trn)
y_val = pd.DataFrame(y_val)
y_trn = std_sclr.fit_transform(y_trn)
y_val = std_sclr.fit_transform(y_val)
X_trn.columns = X.columns
X_val.columns = X.columns
lrm_s = LinearRegression()
lrm_s.fit(X_trn, y_trn)
# X_train = scaler.fit_transform(X_train)
# X_vaid = scaler.transform(X_valid)
# X_test = scaler.transform(X_test)
# model.fit(X_train,y_train)
# y_pred_valid = model.predict(X_valid).reshape(-1,) # array.reshape(-1, 1)
Out[1504]:
In [1506]:
print('\n* SCALED Data: Coefficients:', lrm_s.coef_)
print('\n* SCALED Data: Intercept:', lrm_s.intercept_)
Markdown('###### * SCALED Data: $R^2$ Score for In Sample, Out Of Sample (Validaton dataset: SCALED Data):')
print(' * In Sample R^2:', lrm_s.score(X_trn, y_trn), 'Out Of Sample R^2:', lrm_s.score(X_val, y_val))
print(' * Root Mean Square Error RMSE:', mean_squared_error(y_val, lrm_s.predict(X_val))**0.5)
# print(lrm_s.summary())
Out[1506]:
In [1507]:
# print('Predict y on Validation dataset:', lrm.predict(X_val).reshape(-1,))
# print('Predict y on Train dataset:', lrm.predict(X_trn).reshape(-1,))
In [1519]:
poly = PolynomialFeatures(degree=2, interaction_only=True)
X_trn_p2 = poly.fit_transform(X_trn)
X_val_p2 = poly.fit_transform(X_val)
poly2m = LinearRegression()
poly2m.fit(X_trn_p2, y_trn)
y_pred_poly2 = poly2m.predict(X_val_p2)
# print(y_pred_poly2)
print('* Data SCALED & CLEANED (No Dups, No Outliers):')
print('* R^2 Score: In Sample:', poly2m.score(X_trn_p2, y_trn), 'R^2 Score: Out Of Sample:', poly2m.score(X_val_p2, y_val))
print('* Root Mean Square Error RMSE:', mean_squared_error(y_val, poly2m.predict(X_val_p2))**0.5)
print('* Additional Features/Columns (Polynomials) Created:', X_trn_p2.shape[1] - X_trn.shape[1])
Out[1519]:
In [1520]:
poly = PolynomialFeatures(degree=3, interaction_only=True)
X_trn_p3 = poly.fit_transform(X_trn)
X_val_p3 = poly.fit_transform(X_val)
poly3m = LinearRegression()
poly3m.fit(X_trn_p3, y_trn)
y_pred_poly3 = poly3m.predict(X_val_p3)
# print(y_pred)
print('* Data SCALED & CLEANED (No Dups, No Outliers):')
print('* R^2 Score: In Sample:', poly3m.score(X_trn_p3, y_trn), 'R^2 Score: Out Of Sample:', poly3m.score(X_val_p3, y_val))
print('* Root Mean Square Error RMSE:', mean_squared_error(y_val, poly3m.predict(X_val_p3))**0.5)
print('* Additional Features/Columns (Polynomials) Created:', X_trn_p3.shape[1] - X_trn.shape[1])
Out[1520]:
D1.3 ^^^ Completed & Delivered As Per Above Exhibits: Feature Engineering Techniques: Ends Here: ^^^ :¶
- $D1.3.a:$ DONE: Extract a New Feature and/or Drop a Feature (if required):
- NO New Feature(s) Extracts & NO Feature(s) Drop: As per the above Obervations and Evaluations it is not Recommended to Extract New Features or Drop any Existing Features. We can get High Performace on the Higher side of the Expected Range (As per Delivrable D2.x in the next section below).
- $D1.3.b:$ DONE: Get data model ready and do a train test split
- Created THREE Datasets: TRAIN (X_trn, y_trn), Validation (X_val, y_val), TEST (X_tst, y_tst): To prevent DATA LEAKS, all training & testing to be done on _trn and _val datasets. TEST datasets (_tst) to be RESERVED and kept aside, UNSEEN from any modelling, transformation algorithms activities until the very end when we decide on Final Best Model. At that time we will use the TEST dataset to test and evaluate the FINAL BEST MODEL.
- $D1.3.c:$ DONE: Decide on complexity of the model: Simple Linear, Quadratic or Higher degree
- Performed Polynomial operations for degree/order = 1,2,3 (Linear, Quadratic, Cubic).
- Drastic GAIN in $R^2$ and Drastic Reduction in RMSE to 0.56.
- The max $R^2$ achieved for degree=3 (Cubic) was 84% (in Sample) and 68% (Out of Sample) which is below Expectation.
- Cost: 29 ADDITIONAL Features for Quadratic and 85 ADDITIONAL Features for Cubic. This "Curse Of Dimensionality" is not worth the Performance obtained.
- HENCE: Use SIMPLE LINEAR REGRESSION MODEL: Because we can get BETTER results without Additional / Engineered Features or Dropping Existing Features using other Regression alogrithms like Decision Tree, Random Forest, Boost Algorithms with x_SearchCV otpions (As per Delivrable D2.x in the next section below).
*D2: Creating the Model and Tuning it:*¶
D2.1: vvv Algorithms: Begins Here: vvv¶
Use the Algorithms that you think will be suitable for this project (at least 3 algorithms). Use ***Kfold Cross Validation*** to evaluate model performance. Use appropriate metrics and make a DataFrame to compare models w.r.t their metrics.
Dataset PreProcessing Choices: As per foregoing observations based on analytics and results it appear that:¶
- NO Significant Performance Improvement is achieved with:
- Scalling & Polynomials.
- Presence or Absence of of 25 (2.43%) Duplicated Rows in the dataset of 1030 rows.
- Presence or Absence of of 99 (1.07%) Outliers in a dataset (df) of 1030 rows x 9 cols = 9,270 elements.
- HENCE: To keep the soultion Simpler & Smaller and YET achiving the expected HIGH Performance, we will REVERT back and USE the ORIGINAL dataset having the above mentioned duplicates & outliers.
In [1523]:
# As we just concluded above, we will revert back to the Orifinal df WITHOUT any preprocessing applied for dups, outliers.
# Also we will split dfs again:
# Prepare data for split: Create X, y (Predictor, Predicted) datasets times 3 for Train, Validation, Test
# X = cdf.copy() # Dups & Outliers Removed
X = cdf1.copy() # Contains: 25 duplicates; 99 Outliers, i.e. Before removing Dups & Outliers
y = X.pop('mpa')
# Split df data into 3 datasets: Train, Test, Validate:
X_trn, X_tst, y_trn, y_tst = train_test_split(X, y, test_size=0.2, random_state=3)
X_trn, X_val, y_trn, y_val = train_test_split(X_trn, y_trn, test_size=0.25, random_state=3)
X.shape, y.shape, X_trn.shape, X_tst.shape, X_val.shape, y_trn.shape, y_tst.shape, y_val.shape
type(X_trn), type(X_tst), type(X_val), type(y_trn), type(y_tst), type(y_val)
Out[1523]:
Out[1523]:
In [1524]:
# Trying Decision Tree:
dTree = DecisionTreeRegressor(random_state=6)
dTree.fit(X_trn, y_trn)
print(dTree.score(X_trn, y_trn))
print(dTree.score(X_val, y_val))
Out[1524]:
In [1525]:
# Trying Decision Tree: PRUNED:
dTreeR = DecisionTreeRegressor(max_depth=9, min_samples_leaf=3, random_state=6)
dTreeR.fit(X_trn, y_trn)
print(dTreeR.score(X_trn, y_trn))
print(dTreeR.score(X_val, y_val))
Out[1525]:
In [1526]:
print (pd.DataFrame(dTree.feature_importances_, columns = ["Imp DTree UnPruned"], index = X_trn.columns))
print (pd.DataFrame(dTreeR.feature_importances_, columns = ["Imp DTree Pruned"], index = X_trn.columns))
In [1529]:
# Random Forest Classifier Learning Model:
rfcl = RandomForestRegressor(n_estimators = 50, random_state=6, max_features=8)
rfcl = rfcl.fit(X_trn, y_trn)
# y_predict = rfcl.predict(X_test)
# y_predict_rf = y_predict
# print(rfcl.score(X_test, y_test))
print(rfcl.score(X_trn, y_trn))
print(rfcl.score(X_val, y_val))
In [1530]:
# Build AdaBooster Learning Model:
abcl = AdaBoostRegressor(n_estimators=90, random_state=6, learning_rate=0.6)
abcl = abcl.fit(X_trn, y_trn)
# y_predict = abcl.predict(X_test)
# y_predict_ab = y_predict
# print(abcl.score(X_test , y_test))
print(abcl.score(X_trn, y_trn))
print(abcl.score(X_val, y_val))
In [1531]:
# Gradient Boost Learning Model:
gbcl = GradientBoostingRegressor(n_estimators = 40, random_state=6, learning_rate=0.3)
gbcl = gbcl.fit(X_trn, y_trn)
# y_predict = gbcl.predict(X_test)
# y_predict_gb = y_predict
# print(gbcl.score(X_test, y_test))
print(gbcl.score(X_trn, y_trn))
print(gbcl.score(X_val, y_val))
In [1532]:
# Build Bagging Learning Model:
bgcl = BaggingRegressor(base_estimator=dTree, n_estimators=12, random_state=6)
bgcl = bgcl.fit(X_trn, y_trn)
# y_predict = bgcl.predict(X_test)
# y_predict_bg = y_predict
# print(bgcl.score(X_test , y_test))
print(bgcl.score(X_trn, y_trn))
print(bgcl.score(X_val, y_val))
Perform KFold CV operations on the BEST Performing Algorithms: 1. GradientBoostingRegressor & 2. RandomForestRegressor as below:¶
In [1533]:
kfold = KFold(n_splits=50, random_state=6)
model = GradientBoostingRegressor()
results = cross_val_score(model, X, y, cv=kfold) # , scoring='r2')
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [1534]:
kfold = KFold(n_splits=50, random_state=6)
model = RandomForestRegressor()
results = cross_val_score(model, X, y, cv=kfold) # , scoring='r2')
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
$$ * \ Results: Metrics \ Comparision \ * $$¶
| Regression Algorithm | Other Metrics | R^2 Train DataSet | R^2 Validation Dataset | Comments |
|---|---|---|---|---|
| Simple Linear | RMSE: 8.898078241713177 | 0.7330438038819633 | 0.680319427895178 | Data: UnScaled, Cleaned |
| Simple Linear | RMSE: 0.5638396157514333 | 0.7330438038819633 | 0.682084887709276 | Data: Scaled, Cleaned |
| Polynomial LinRegr: Degree=2 | RMSE: 0.5258500933378105 | 0.7922581527934405 | 0.7234816793366159 | 29 New Features Added. Data: Scaled, Cleaned |
| Polynomial LinRegr: Degree=3 | RMSE: 0.563631653160918 | 0.8414541656579173 | 0.6823193595550907 | 85 New Features Added. Data: Scaled, Cleaned |
| Decision Tree, UnPrunned | 0.9999571857586979 | 0.8072407565692836 | Original Data: UnScaled, Not PreProcessed: Contains 25 Duplicate Rows; 99 Outliers | |
| Decision Tree, Prunned: Depth=3 | 0.943778925044059 | 0.7649833982162988 | ^^^ Ditto ^^^ | |
| Random Forest | 0.9865892384207504 | 0.8997314457247279 | ^^^ Ditto ^^^ | |
| AdaBoost | 0.8267219994089168 | 0.8000375936309319 | ^^^ Ditto ^^^ | |
| Gradient Boost | 0.9639792909595184 | 0.90229938952186 | ^^^ Ditto ^^^ | |
| Bagging | 0.9837672362553681 | 0.8981940858966639 | ^^^ Ditto ^^^ | |
| KFold for GradientBoostingRegressor | Accuracy: 89.403% (5.606% Std.Div.) | ^^^ Ditto ^^^ | ||
| KFold for RandomForestRegressor | Accuracy: 91.229% (5.695% Std.Div.) | ^^^ Ditto ^^^ | ||
D2.1 ^^^ Completed & Delivered As Per Above Exhibits: Algorithms: Ends Here: ^^^ :¶
- DONE: Regression Algorithms Used (At Least 3):
- Simple Linear Rgression
- Decesion Tree
- Random Forest
- Ada Boost
- Gradient Boost
- Bagging
- DONE: KFold CV Used for these Best Performing Algorithms: 1. GradientBoostingRegressor 2. RandomForestRegressor
- DONE: Metrics Dataframe/Tabulation: Created & Presented above ^^^
In [1536]:
### My Housekeeping: Jupyter Code File Incremental Backup 8 Sat.Jul.11 12:25am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 8")
# ! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 8.ipynb"
! ls -l Project*.ipynb
Out[1536]:
vvv * D2.2: Tuning Techniques: Begins Here: vvv¶
Employ Techniques to squeeze that extra performance out of the model without making it over fit. Use ***Grid Search or Random Search*** on any of the two models used above. Make a DataFrame to compare models after hyperparameter tuning and their metrics as above.
In [1420]:
# Instantiate a model for RandomForestRegressor:
model_rgr = RandomForestRegressor(n_estimators=50)
# specify parameters and distributions to sample from
param_dist = {"max_depth": [3, None],
"max_features": sp_randint(1, 7),
"min_samples_split": sp_randint(2, 11),
"min_samples_leaf": sp_randint(1, 11),
"bootstrap": [True, False],
"criterion": ["mse", "mae"]}
# run randomized search
# n_iter = number of random samples
randomCV = RandomizedSearchCV(model_rgr, param_distributions=param_dist, n_iter=10) #default cv = 3
In [1421]:
# Fit / Run RandomizedSearchCV for RandomForestRegressor :
randomCV.fit(X, y)
print(randomCV.best_params_)
len(randomCV.cv_results_['mean_test_score']) # MSB: These many model fits (runs)
randomCV.cv_results_['mean_test_score'] # :MSB
Out[1421]:
Out[1421]:
Out[1421]:
In [1436]:
# Instantiate a model for GradientBoostingRegressor:
model_gbr = GradientBoostingRegressor()
# specify parameters and distributions to sample from
param_dist = {"max_depth": [3, None],
"max_features": sp_randint(1, 7),
"min_samples_split": sp_randint(2, 11),
"min_samples_leaf": sp_randint(1, 11),
"learning_rate": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
"criterion": ["mse", "mae", 'friedman_mse']}
# run randomized search
# n_iter = number of random samples
randomCV = RandomizedSearchCV(model_gbr, param_distributions=param_dist, n_iter=10) #default cv = 3
In [1437]:
# Fit / Run RandomizedSearchCV for GradientBoostingRegressor :
! date
randomCV.fit(X, y)
! date
print(randomCV.best_params_)
len(randomCV.cv_results_['mean_test_score']) # MSB: These many model fits (runs)
randomCV.cv_results_['mean_test_score'] # :MSB
Out[1437]:
Out[1437]:
Out[1437]:
In [1441]:
# Based on RandomSearchCV above^ the following is BEST Model / Algorithm with these params:
# Build the FINAL Model & validate it on the UNSEEN / RESERVED TEST dataset:
# Gradient Boost Learning Model:
gbcl = GradientBoostingRegressor(criterion='mse', learning_rate=0.6, max_depth=3, random_state=3,
max_features=6, min_samples_leaf=10, min_samples_split=2)
gbcl = gbcl.fit(X_trn, y_trn)
# y_predict = gbcl.predict(X_test)
# y_predict_gb = y_predict
# print(gbcl.score(X_test, y_test))
print(gbcl.score(X_trn, y_trn))
print(gbcl.score(X_val, y_val))
In [1443]:
# *** FINAL SCORE ON the UNSEEN TEST DATASET: ***
print(gbcl.score(X_tst, y_tst))
In [1442]:
# Perform KFold on this FINAL BEST MODEL to check the Accuracy:
kfold = KFold(n_splits=50, random_state=7)
model = GradientBoostingRegressor(criterion='mse', learning_rate=0.6, max_depth=3, random_state=3,
max_features=6, min_samples_leaf=10, min_samples_split=2)
results = cross_val_score(model, X_trn, y_trn, cv=kfold) # , scoring='r2')
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
D2.2 ^^^ Completed & Delivered As Per Above Exhibits: Tuning Techniques: Ends Here: ^^^ :¶
- Used RandomizedSearchCV for the Two Best Performing Algorithms for the Best Parameter Search:
- RandomForestRegressor: Average Score: 85%
- GradientBoostingRegressor: Average Score: 90%
- BEST Algorithm based on Search Parameters:
- GradientBoostingRegressor(criterion='mse', learning_rate=0.6, max_depth=3, random_state=3, max_features=6, min_samples_leaf=10, min_samples_split=2)
- Test Data: SCORE: 0.9161131402242251 ***(91.61%)***: On the UNSEEN / Reserved Test Data : BEST Final SCORE on high end of Goal Range: 80% to 95%
- KFoldCV Accuracy: 89.826% (7.118% Std.Div) On GradientBoostingRegressor with the Finalized PARAMETERS as above^
In [1449]:
### My Housekeeping: Jupyter Code File Incremental Backup 7 Fri.Jul.10 07:59pm ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 7")
# ! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 7.ipynb"
! ls -l Project*.ipynb
Out[1449]:
In [1538]:
### My Housekeeping: Jupyter Code File Incremental Backup 9 Sat.Jul.11 01:15am ^^^
Markdown("### Incremental Jupyter Notebook Code Backup 9")
! cp "Project 4 FEMST Concrete Strength Predict.ipynb" "Project 4 FEMST Concrete Strength Predict Backup 9.ipynb"
! ls -l Project*.ipynb
Out[1538]: